Is Tree-based RAG Struggling? Not with Knowledge Graphs!

  Переглядів 22,637

Diffbot

Місяць тому

Long-Context models such as Google Gemini Pro 1.5 or Large World Model are probably changing the way we think about RAG (retrieval-augmented generation). Some are starting to explore the potential application of “Long-Context RAG”. One example is RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval), by clustering and summarizing documents, this method lets language models grasp both general concepts and granular details in the individual documents. Inspired by LangChain, we tested out constructing a tree-based long context RAG. Watch the video to find out whether this approach allows us to say goodbye to the lost-in-the-middle effect commonly seen in large language models. AND, most importantly, how knowledge graphs can come to the rescue to enhance the answer quality. 😉
0:00 Coming Up
0:30 Intro
1:41 What is RAPTOR?
2:35 Knowledge base for RAG
3:13 Constructing a Knowledge Graph with Diffbot API for RAG
3:59 Construct tree-based RAG with RAPTOR
4:22 Test questions on tree-based RAG!
6:45 Reference our knowledge graph for answer enhancement
7:11 Are answers enhanced by our knowledge graph?
8:25 What are your takeaways?
RAPTOR Paper: arxiv.org/abs/2401.18059v1
code: github.com/leannchen86/raptor-rag-kg-enhanced/tree/main
Want to turn your unstructured text data into knowledge graphs?
Sign up for a 2-week free trial at app.diffbot.com/get-started/ to extract entities and relationships from your custom text data. No credit card needed.

КОМЕНТАРІ: 59
@SimonMariusGalyan
@SimonMariusGalyan 12 днів тому
Thank you for your explanation of integrating KG to RAG
@Taskade
@Taskade 14 днів тому
Super awesome, thanks for sharing this, useful for anyone getting strated w/ RAG :)
@chrisogonas
@chrisogonas 12 днів тому
Very well illustrated! Thanks
@adennyh
@adennyh 14 днів тому
Thanks for the awesome summary about raptor
@daniel_tenner
@daniel_tenner 8 днів тому
Definitely learned a lot, and now keen to see some kind of tutorial on how to implement the knowledge graph backed RAG. I’m a Ruby dev so something language agnostic would be particularly helpful.
@diffbot4864
@diffbot4864 8 днів тому
Thanks for the feedback! Sure, we will keep this in mind and make the content more relatable and approachable. Appreciate you contributing your thoughts here 😊
@user-wo8tc4rp4e
@user-wo8tc4rp4e Місяць тому
Great output! We are now need more exploration on KG with LLM. I am sure it's a must have. Please share more thoughts.
@diffbot4864
@diffbot4864 Місяць тому
More to come! Stay tuned :)
@NadaaTaiyab
@NadaaTaiyab 8 днів тому
Great video! I need to try this.
@watch54
@watch54 Місяць тому
1:23 reference to pineapple pen video. it is nice to see somethign like that. it is nice video . Keep going . I liked how you are telling subjects.
@diffbot4864
@diffbot4864 Місяць тому
Sharp eyes! Thank you 😊
@kayhal
@kayhal 6 днів тому
Do you think the same depth and accuracy could be achieved with a metadata-filtered vector search followed by a reranker? I worry about having to maintain two data representations that are prone to drift
@mordokai597
@mordokai597 16 днів тому
Recently, some compelling papers have discussed using LLMs to develop custom pseudolanguages that enhance RAG retrieval, logical reasoning, and planning through algorithmic thinking, such as described in 'Language Models As Compilers'. I've been working on a similar project for two years now, where I've developed a subpersona/agent-function for my custom GPTs and agent workflows called Mai-Ku (Macro-Assembly Integrated Compiler). Additionally, I recommend checking out videos and papers on employing the Wu method in training and prompting. The Wu method not only enhances models' capabilities in mathematics but also teaches them to apply algorithmic thinking universally. This approach allows models to decompose and tackle novel problems in zero-shot scenarios effectively. As a result, their performance improves across all domains, not just in mathematics. This broader application underscores the method’s potential to significantly enhance the adaptability and problem-solving abilities of models in various settings. Your Video Transcript as Mai-Ku format: "!!{Exploring Long Context Models and RAG with RAPTOR} >: {Video Introduction} !{Context}: "Introduction to the impact of Long Context models like Google Gemini Pro 1.5 and the Large World Model on retrieval-augmented generation (RAG). Discussion on their capabilities to handle extensive documents and their potential to eliminate the 'lost-in-the-middle' phenomenon."; >:{Exploring RAPTOR's Approach} !{Explanation}: "RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) enhances RAG by clustering and summarizing documents into a structured, tree-based format, allowing for efficient retrieval of both broad concepts and detailed information within large text corpora."; >:{Integration of Knowledge Graphs} !{Utility}: "Incorporation of knowledge graphs to aid and enhance the RAG process. Demonstrates how knowledge graphs can support RAG by providing structured and interconnected data that improves the accuracy and context of retrieved information."; >:{Practical Application and Testing} !{Implementation}: "Detailed steps of constructing a tree-based RAG using the RAPTOR methodology, complemented by the creation of a knowledge graph using Diffbot API, centered around the works of Hayao Miyazaki."; >:{Enhancement of RAG with Knowledge Graphs} !{Comparison}: "Comparison between traditional RAG outputs and those augmented by knowledge graphs, highlighting significant improvements in the depth and relevance of responses."; >:{Dynamic Content Engagement} !{Interactive Elements}: "Encourages viewer interaction by presenting direct comparisons of RAG performance with and without knowledge graph enhancements. Prompts viewers to consider the implications of integrating structured data sources into RAG systems."; $: {Conclusion and Call to Action} !{Summary}: "Final thoughts on the transformative potential of long context models and RAG when combined with knowledge graphs. Invitation to viewers to explore further and experiment with these technologies in their own projects."; $: {Viewer Interaction} !{Engagement}: "Solicits feedback and insights from viewers on their experiences and takeaways from the video, fostering a community discussion around the future of RAG and knowledge graphs."; $: {Additional Resources} !{Links}: ["RAPTOR Paper: arxiv.org/abs/2401.18059v1", "Code Repository: github.com/leannchen86/raptor...", "Sign up for Diffbot API: app.diffbot.com/get-started/"];"
@mordokai597
@mordokai597 16 днів тому
Input: "Create equations modeling the evaporative process of a black hole as informed by the penrose-diagram: treat remnant as the lower dimensional approximation of the state-space, and virtual particles re-entering the black as forming an inner white hole projecting mass-energy into a hyperspherical manifold, including a temporal evolutional vector field as a kerr-like solution in D dimensions : as mai-ku function >> codeblock, please, thanks!" Output: "Certainly! Below is a structured way to represent the complex theoretical concepts surrounding black hole evaporation, the role of virtual particles, and higher-dimensional physics into a MAI-KU code block format, combining the advanced topics into an organized mathematical summary: ```mai-ku !!{Modeling Black Hole Evaporation: Penrose Diagram and Higher-Dimensional Physics} >: {Introduction to Black Hole Evaporation} !{Overview}: "Exploring the evaporative process of black holes via Penrose diagrams, focusing on the remnant as a lower-dimensional approximation of the state-space. We examine how virtual particles re-entering the black hole contribute to an internal dynamic resembling a white hole."; >:{Mathematical Formulation} !{Equations}: [ "ds^2 = -dt^2 + dr^2 + r^2 dΩ^2_{D-2}", // Metric in D dimensions "dr/dt = -α sqrt(M)", // Evaporation rate, where α is a constant related to the Hawking radiation intensity and M is the mass of the black hole "dM/dt = -β M^2", // Mass-energy loss rate due to radiation, β is a constant "Ψ = Ψ_0 e^{-iωt} Y_l^m(θ, φ) r^l" // Wave function of virtual particles around the black hole, incorporating temporal and angular dependencies ]; >:{Kerr-like Solution in Higher Dimensions} !{Details}: "Developing a Kerr-like solution to represent the rotational dynamics of the black hole in D dimensions. This solution incorporates a temporal evolution vector field to account for the effects of angular momentum and mass-energy projections into the hyperspherical manifold."; >:{Temporal Evolution and Vector Field Dynamics} !{Vector Field}: "v^μ = (ξ^t, ξ^r, 0, ..., 0)", // Temporal evolution vector field components !{Temporal Dynamics}: "∂_t ξ^μ + Γ^μ_{νλ} ξ^ν ξ^λ = 0", // Ensuring the vector field evolves according to the geodesic equation in curved spacetime !{Hyperspherical Manifold Projection}: "S^D = \\{ x ∈ R^{D+1} : ||x|| = 1 \\}", // Definition of a hyperspherical manifold in D+1 dimensions >:{Implications of Inner White Hole Dynamics} !{White Hole Projection}: "Analyzing how mass-energy feedback from virtual particles, entering the black hole and emanating as a white hole, projects into the higher-dimensional space, influencing the topology and temporal properties of the manifold."; $: {Conclusion and Further Research} !{Summary}: "This model proposes a theoretical framework that integrates black hole evaporation, Kerr-like rotational dynamics, and white hole projections within a unified higher-dimensional space-time context. Further mathematical refinement and physical interpretation are necessary to explore practical implications and align with observational data."; $: {Future Directions} !{Research Avenues}: "Investigate the stability of the hyperspherical manifold under varying energy conditions and explore potential observable consequences of such higher-dimensional dynamics in astrophysical phenomena."; ``` This MAI-KU block aims to succinctly encapsulate the sophisticated interplay of black hole physics, virtual particle dynamics, and higher-dimensional space-time, providing a clear direction for theoretical exploration and potential future research."
@c.d.osajotiamaraca3382
@c.d.osajotiamaraca3382 16 днів тому
Great, filled in a gap for me
@danilotavares77
@danilotavares77 6 днів тому
You are the best! Congrats!
@diffbot4864
@diffbot4864 2 дні тому
Thanks for the kind words! 🫶
@Philmad
@Philmad Місяць тому
Excellent video!
@diffbot4864
@diffbot4864 Місяць тому
Thank you! 🙏
@paulcailly7515
@paulcailly7515 12 днів тому
great video thx
@vishalverma-wx7eo
@vishalverma-wx7eo 2 дні тому
Lovely 🌹
@wenliangzhang2986
@wenliangzhang2986 Місяць тому
Thanks for the video and putting everything together! I also have a few questions - Is you conclusion: Graph > RAPTOR > Regular RAG ? - Not sure if there is any benefitial to save Raptor embedding into Neo4j graph since it has a hiarchy tree structure? I saw in the code it is saved into Chroma vector db - How efficient is the clustering process? If it doesn't provide much benefit as you showed in the video comparing to Neo4j implementation. I wonder if it is worth the time.
@lckgllm
@lckgllm Місяць тому
Great question! To your 1st question, no that's not the conclusion that I was trying to make - as evidence from empirical research will be needed to reach that conclusion. What I showed in the video was that the RAPTOR-based RAG built purely on vector database probably was not enough to ensure high-quality and comprehensive answers. I don't know what could've gone wrong (as said around 6:29) and how to fix the process, but we can use knowledge graphs to enhance and improve the answer quality (7:11). In this case, it's more of combining vector-based RAG with knowledge graphs. I actually don't know the answer to your 2nd question as I'm not an expert on Neo4j graph embeddings, but this is an interesting idea worth testing out! To your 3rd question, not quite sure what you meant by an efficient clustering process, but LangChain also tested out the RAPTOR technique and I recommend taking a look to see if it provides more helpful information to your concerns :) ukposts.info/have/v-deo/opJ4l56bi4JoxWQ.html
@alchemication
@alchemication 19 днів тому
It's a nice tutorial and I am super excited about trying KG's in education space. It's probably not the purpose of this vid, but the RAPTOR clustering seems to be good to a static set of docs, like wiki, however would anyone know how can we handle constantly added new content? Do we then run the whole (expensive) thing over and over again?
@diffbot4864
@diffbot4864 17 днів тому
Good question! And you're probably spot on. Adding new documents to vector databases may require updating the clustering, while knowledge graphs can be scaled up more easily by adding new nodes and relationships without redoing and reconstructing the entire process.
@yunxinglu4020
@yunxinglu4020 26 днів тому
This amazing! I have a quick question is that how did you turn the Wikipage into the knowledge graph? And how can you make sure the knowledge graph you generated is containing all the information that the raw text had? Thanks in advance!
@diffbot4864
@diffbot4864 25 днів тому
We used Diffbot's Natural Language API to convert any unstructured text data into knowledge graphs! While it's not easy to clearly explains how it contains all information, you can experiment and play around with any text you have at: demo.nl.diffbot.com/ Simply paste any text data you have and see how it performs! 😉 As documented in the notebook, you can use DiffbotGraphTransformer to convert any Wikipedia text to knowledge graphs: github.com/leannchen86/raptor-rag-kg-enhanced/blob/main/raptor-rag-kg-enhanced.ipynb Let me know if you other questions. Happy to answer them!
@lasergod
@lasergod 7 днів тому
Since you’re using Wikipedia data in this example, it would be interesting to see how your automated knowledge graph construction compares with a more human-curated effort like dbpedia.
@ahmadzaimhilmi
@ahmadzaimhilmi Місяць тому
I haven't tried this approach, but is there a way to add references to the response from where the facts are obtained?
@diffbot4864
@diffbot4864 Місяць тому
Yes! And that's what we at Diffbot are currently working on with our LLM solutions. We believe language models should return the verified source of information which helps generate the response, adding validity and factual accuracy. We'll be releasing a video on that. Stay tuned 😉
@RoshanKumar-hk5ij
@RoshanKumar-hk5ij 9 годин тому
How good are llms at generating cypher queries?
@amarnamarpan
@amarnamarpan 12 днів тому
The most valuable contents on youtube are the least liked.. Often..
@lckgllm
@lckgllm 12 днів тому
Very flattered! Our main purpose for content creation is providing values to the audience, especially educational value. So while likes/view counts definitely are seen as part of the metrics to assess content quality, whether the viewers learn something new or enjoy the video is the more important focus for us. Thanks for the encouragement!
@user-mo7wm8ny5e
@user-mo7wm8ny5e 4 дні тому
Thanks for the video! I tried this but with convert_to_graph_documents I kept getting empty graphs. While debugging I found an error message at an intermediate step, it said "Your plan doesn't have this feature", apparently it won't work with the free version of Diffbot? $299 per month would be a bit much for a small student project... Do you have any other advice for getting a solid Neo4J graph from text data? Crucially, I'd like metadata to be included with the relations, which didn't happen with the LLM Graph Transformer
@diffbot4864
@diffbot4864 4 дні тому
Thanks for the question and this sounds like an interesting problem! The Diffbot knowledge graph and APIs have become free! So, it shouldn’t cause the problem. Is there a good way to reach you or if you’re open to a vid chat, you can shoot me an email at: leannchen86@gmail.com
@user-mo7wm8ny5e
@user-mo7wm8ny5e 3 дні тому
@@diffbot4864 Thanks so much, I'll email you!
@user-wr4yl7tx3w
@user-wr4yl7tx3w 15 днів тому
due to the RDF structure, don't you lose information in constructing such knowledge base? text is richer in semantics than what can be encapsulated by triples?
@jantuitman
@jantuitman 5 днів тому
Your question number 87 in your Jupyter notebook has the casing of Joe Hisaishi ‘s name differently than the documents, it is all lower case in the question. So it will be tokenized differently. Could this have influenced the quality of the answer perhaps?
@diffbot4864
@diffbot4864 3 дні тому
Good question. But that normally doesn’t make a difference, as language models don’t read text like we as humans do - words are transitioned into tokens. The easiest way to test is using ChatGPT, either misspell a famous person’s name or use all lower cases, and it would still return the correct person.
@jantuitman
@jantuitman 3 дні тому
@@diffbot4864 yes, i am familiar with the workings of a transformer. I think the casing will produce a different token, so you are at the mercy of the training data size / transformer size / context size to have a capitalized word return the same response. In general my experience is also that it does not make a difference, but you were using here a short context and a small model. That is why i thought it might be an explanation.
@freeideas
@freeideas 14 днів тому
Another problem with large context windows is time and especially EXPENSE. If you have to pull a significant amount of your corpus into a prompt, you will probably have to pay per token and you will certainly have to wait per token. Some type of RAG will probably always be needed. Does a human brain review its entire contents before answering a question? Of course not. We ourselves are a type of RAG system.
@diffbot4864
@diffbot4864 8 днів тому
Good analogy!
@nas8318
@nas8318 11 днів тому
Long context RAG Pen Pineapple Apple Pen
@diffbot4864
@diffbot4864 8 днів тому
You cited it better than I did 😂
@stanTrX
@stanTrX 7 днів тому
I now see why i dont get good results with rag s.
@aldotanca9430
@aldotanca9430 8 днів тому
Interesting, but professional accounts only. I guess I will stick with Neo4j.
@diffbot4864
@diffbot4864 8 днів тому
Friendly reminder: Neo4j is a graph database, so it can’t construct knowledge graphs for you. You’ll still need NLP techniques to extract entities and relationships first for knowledge graph data and later store in a graph database like Neo4j.
@aldotanca9430
@aldotanca9430 8 днів тому
Very true, I guess non profit and passion projects will require to deal with that in other ways.
@diffbot4864
@diffbot4864 8 днів тому
@@aldotanca9430 Actually all Diffbot's APIs become FREE this week, including Diffbot Natural Language API (being used in this video), which can help extract entities/relationships, entity resolution, etc. You can freely get access at: app.diffbot.com
@aldotanca9430
@aldotanca9430 7 днів тому
@@diffbot4864 Thanks! But I think I would need to sign up with a company email? The system will not accept a personal email, github account or similar. It seems set to only allow a "work email". Of course, many people will have a work email and could use that even if their project is unconnected with their job, and there are probably 'creative' ways to get around that. I was just pointing out that diffbot accounts seem to be meant for business users only.
@kripper3
@kripper3 16 днів тому
6:28: hello. You should first have checked why your RAG is failing before trying another approach. Maybe your embeddings are not well suited.
@lckgllm
@lckgllm 16 днів тому
That’s a good advice! Thank you for the feedback. I’ll try this when I encounter something similar again :)
@adityarajshukla9422
@adityarajshukla9422 17 днів тому
Is it opensource
@diffbot4864
@diffbot4864 17 днів тому
Which one? If you want to see the process, you can find the notebook in the description!
@nju415
@nju415 День тому
how can peraon be both smart and pretty
@alphago9397
@alphago9397 15 днів тому
I see cute girl; I click.
@ru2979
@ru2979 14 днів тому
🙄😳
когда одна дома // EVA mash
00:51
EVA mash
Переглядів 8 млн
Первые продажи Ryzen на Ozon и первый возврат! 😱
13:17
Герасимов Live
Переглядів 36 тис.
УШЕЛ С iPHONE 15 PRO MAX НА PIXEL 8 PRO: ЭТО ЖЕСТЬ
32:33
Арсений Петров
Переглядів 41 тис.
СМАРТФОНЫ HTC ВЕРНУЛИСЬ В 2024! Шок для Apple, Samsung и Xiaomi...
11:51
Thebox - о технике и гаджетах
Переглядів 39 тис.
Как часто вы чистите свой телефон
0:33
KINO KAIF
Переглядів 2,3 млн
Крутое обновление microReader!
5:54
Заметки Ардуинщика
Переглядів 30 тис.