When designing an information architecture, you must approach the system — a website, product, book, whatever — as a whole. For example, to design a site’s global navigation scheme, you must understand what kinds of things will be in the site and how they differ from each other. You can’t do it by examining individual content items: you must grok all the site’s content.

I’ve written previously about how AIs can augment information architecture work. Alas, out-of-the-box, LLMs work more effectively with individual content items than with larger sets, such as entire websites. You could say they’re great for working with trees — or even individual leaves, branches, and roots — but less effective when working with entire forests.

For example, in my AI tagging experiment, the LLM ‘read’ each post one by one, assigning it the three most relevant terms from a predefined taxonomy. The LLM never got a sense of the site as a whole; all it saw was individual posts. This worked because a) my blog doesn’t cover a lot of ground, and b) I started with a predefined taxonomy that accurately described that ground.

But what if that wasn’t the case? What if we wanted to work on an entire website rather than on one post at a time? That requires different techniques. This post explains the use of one such technique: retrieval augmented generation (RAG) using knowledge graphs.

Introducing Graph RAG

At this point, you likely understand how generative AI works, more or less: you issue a prompt and get a response back. The accuracy of the response depends on whatever the AI ‘knows’ about what you’re asking. For example, if you want a summary of War and Peace, you can just type that as a prompt and get a good result. This book is old and famous; its full text is likely included in the LLM’s training corpus.

That’s not the case for a report produced by someone in your team. If you ask for a summary of the report, the AI won’t know what you’re referring to. To get the desired result, you must pass the report along with the prompt. Or consider another common LLM use case: spellchecking a text. That obviously won’t work unless you include the text along with the request.

Which is to say, for some use cases, you must tell the LLM what data to work with. Providing more relevant content along with the prompt reduces hallucinations, leading to better results. The more content (and context) you provide, the better.

Different models have different ‘context windows’ (roughly, how much data they can work with in each request.) More sophisticated models have larger context windows, meaning that you can work with more data. Still, there are limits to how much you can send (and receive) at a time.

One approach to overcoming these limitations is a technique called retrieval augmented generation, or RAG. The basic idea is searching within a predefined content repository for appropriate data to inject into prompts to make them more relevant.

For example, my publisher (Rosenfeld Media) offers a system called Rosenbot that lets users ask questions and have discussions about the books in the RM catalog. If you ask a question about note-taking, you’ll see responses influenced by my book Duly Noted, whose text is included in the Rosenbot’s content repository.

The technique is similar to traditional search: when the user asks a question or issues a command, the system looks up relevant text within the repository and adds it to the prompt. The additional context provided by this extra text makes a big difference, focusing responses on the material available in the repository.

By default, this material consists of unstructured text. But text can also be structured to express semantic meanings, reducing ambiguity. These relations are captured in a knowledge graph, a structure that captures relationships between ideas.

For example, the verb save means different things when used in a bank, church, or computer lab. These meanings aren’t inherent in the word itself; they only become clear when it’s used alongside other terms. A knowledge graph might specify that customers save money. “Customers” and “money” are two types of things, and “saving” is something one does to the other. By relating them to each other in this way, we specify a particular meaning to the verb save.

You may see where this is going. In theory, you could get better results by using RAG with knowledge graphs rather than plain text. That’s exactly what graph RAG does. And it’s not just theory.

A Real-world Example

Microsoft recently released an open source framework appropriately called GraphRAG, which it describes as “a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using the power of LLMs.” I.e., it turns unstructured text into knowledge graphs for use in RAG.

To illustrate, I used GraphRAG to work with the full set of transcripts from my podcast. Rather than focusing on individual episodes (as I did with my blog,) this allowed me to work with the entire set of interviews as a whole. For example, I asked GPT to identify themes that have emerged on the podcast over time. This is what it responded:

Common Themes in the Transcripts

Community and Collaboration

A recurring theme across the transcripts is the importance of community and collaboration. Various contexts, including technology, education, and professional networks, emphasize the value of collective efforts and shared knowledge in driving innovation and improving outcomes. Entities like the Open Source Community, San Francisco’s tech ecosystem, and the User Experience Design Community highlight the significance of collaborative efforts and knowledge sharing to enhance productivity and creativity [Data: Reports (720, 714, 764, 1101, 592, 1142, 557, 1485, 893, 1231, +more)].

Role of Technology

The transformative impact of technology on various fields is another prominent theme. Technology’s dual role in enhancing and challenging fields such as urban planning, teenage girls’ reading habits, and visual communication is highlighted. The integration of AI, mobile devices, and digital tools in various sectors underscores the importance of technology in optimizing operations and improving user engagement [Data: Reports (973, 711, 565, 797, 842, 1411, 603, 801, 97, 1232)].

Professional Development and Networking

Professional development and networking are emphasized as crucial for career growth and knowledge sharing. Platforms like LinkedIn and events such as Taxonomy Bootcamp and World Information Architecture Day serve as central hubs for professional growth and networking. These platforms and events facilitate the exchange of ideas and foster professional relationships [Data: Reports (741, 1293, 371, 1007, 794, +more)].

Leadership and Mentorship

Leadership and mentorship are highlighted as essential for professional growth. Key figures like Donna in the Evidence-Based Coaching Community and Eduardo Ortiz in the &Partners Community leverage their experience to guide and develop others. The significance of mentorship in fostering professional development and advancing knowledge is a recurring theme [Data: Reports (641, 1194, 587, 524, 1287)].

Effective Communication

Effective communication strategies are crucial, particularly in behavioral communication, trust-building, and audience engagement. Discussions about Rob’s expertise in behavioral communication, Crutchfield’s trust-building practices, and the role of ESL teachers in enhancing communication underscore the importance of clear and effective communication in various contexts [Data: Reports (1117, 203, 729)].

Strategic Thinking

The integration of strategic thinking with various fields such as design, technology, and organizational practices is a common theme. Discussions about Nathan’s role in design and strategy, Simon’s Wardley Mapping, and the use of Go in AI decision-making highlight the importance of strategic clarity and adaptability in achieving organizational objectives [Data: Reports (521, 1366, 550)].

User Experience (UX)

User experience (UX) is a recurring theme, emphasizing its importance in design, usability, and content strategy. Various entities like Open Source Projects, Visual Design, and notable figures such as Brin and Page contribute to the evolution of UX, highlighting its relevance in enhancing user satisfaction and achieving business objectives [Data: Reports (698, 876, 809, 674, 548, +more)].

Inclusivity and Accessibility

The importance of inclusivity and accessibility in design and technology is frequently mentioned. Communities focused on cognitive accessibility, neurodiversity, and civic design emphasize the need for inclusive approaches that cater to diverse user needs. This theme highlights the role of adaptive technologies and inclusive practices in creating equitable experiences [Data: Reports (964, 947, 652, 567, 589, 250, 1471)].

Knowledge Management

Effective information and knowledge management is a recurring theme. Various entities emphasize the significance of organizing and managing information efficiently in the digital age. Tools and methodologies like the Zettelkasten method, MusicBrainz, and personal knowledge management systems are highlighted for their roles in enhancing productivity and information management [Data: Reports (1354, 1163, 1338, 1241, 374)].

Ethical Considerations

Ethical considerations and social responsibility are emphasized, particularly in the context of design, technology, and organizational practices. Discussions often focus on the ethical implications of decisions and the need for responsible practices to address societal challenges and ensure equitable outcomes [Data: Reports (1493, 1161, 676, 984)].

These themes collectively underscore the interconnectedness of community, technology, leadership, and ethical practices in shaping various fields and driving innovation. The emphasis on collaboration, strategic thinking, and effective communication highlights the multifaceted approach required to navigate and excel in today’s dynamic environments.

As of today, there are 144 episodes in my podcast. All have transcripts. Reading through everything would’ve taken me a long time, and I wouldn’t have done as good a job at spotting these patterns as GPT did. But I also wouldn’t have been able to pass all of them in one go to the LLM; the context window isn’t large enough.

This list gave me a ‘big picture’ understanding of things we’ve discuss in the podcast over time. But I could also get more granular if I wanted. Note that the list above gives references to specific episodes. For example, I followed up by asking which episode covered evidence-based coaching and who was the guest, and GPT responded with Donna Lichaw in episode 7.

This zooming in and out between details and the big picture is central to my work as an information architect. With graph RAG, I can run LLM operations on the whole set of documents or focus just on one specific idea. It’s akin to having a powerful lens that covers a wide focusing range, from macro zoom to a wide landscape POV.

Downsides

Is it perfect? No; the system makes rookie mistakes. (But, read caveat below: this may be my fault for being cheap.) Still, I wouldn’t want an LLM categorizing things straight into production without human oversight. For example, I asked it to write a taxonomy of categories used to describe the content of these interviews. This is what it came up with:

Taxonomy of Categories for Interview Content

1. COVID-19 Impact

  • Public Health Measures: Mask-wearing, social distancing, treatment options, Long COVID [Data: Reports (1505)].

2. Design and Strategy

  • Design Practices: Foodicons Project, design education, integration of design and strategy [Data: Reports (521, 1366, 1280)].
  • User Experience and Design: UX principles, visual design, open source projects, key figures in design [Data: Reports (698, 547, 1114, 749, 666, 1435)].
  • Collaborative Design: Platforms like MURAL, design thinking, remote work settings [Data: Reports (1068, 1019, 1156, 1054, 684)].

3. Strategic and Organizational Development

  • Company Evolution: Strategic tools, team dynamics, market adaptation [Data: Reports (1261, 445, 761, 334, 1099, 1108)].
  • Organizational Initiatives: Decision-making, communication, design practices [Data: Reports (569, 911)].

4. Technology and Innovation

  • Technological Advancements: Augmented reality, virtual reality, AI impact on jobs [Data: Reports (1088, 661, 292, 880, 1307, 640)].
  • Digital Transformation: Internet, DevOps, mobile technology [Data: Reports (830, 1201, 979, 726)].

5. Professional Development

  • Career Growth: Skill acquisition, career architecture, professional development coaching [Data: Reports (1446)].
  • Professional Networking: LinkedIn, career development, job searching [Data: Reports (741)].

6. Knowledge Management and Productivity Tools

  • Software and Tools: Tinderbox, Scrivener, DEVONthink [Data: Reports (1328, 1355)].
  • Personal Knowledge Management: Apple Notes, Zettelkasten method, Roam, Obsidian [Data: Reports (1354, 1163, 1338, 1119)].

7. Community and Collaboration

  • Community Engagement: Tampa Tech Community, InCitu’s AR tools [Data: Reports (739, 382)].
  • Collaborative Platforms: Google Docs, YouTube Channels, Microsoft Docs [Data: Reports (1196, 1212, 1391)].

8. Historical and Cultural Impact

  • Significant Events: Rwandan Genocide, Protestant Reformation [Data: Reports (1049, 847)].
  • Cultural Initiatives: Ubumuntu initiative, TechsMechs event [Data: Reports (1056, 704)].

9. Healthcare and Public Safety

  • Public Health Institutions: CDC, WHO, health crisis management [Data: Reports (866)].
  • Medical Technology: 3M healthcare technology, leukemia patient experiences [Data: Reports (446, 1239)].

10. Educational Practices and Institutions

  • Teaching Methodologies: UC Berkeley’s Center for Teaching and Learning, graphic organizers [Data: Reports (1178, 1337, 796)].
  • Professional Development: Workshops, intensives, skill development [Data: Reports (1183, 196, 1448)].

11. Cognitive Science and Personal Development

  • Cognitive Tools: Memory Palace, Polarity Mapping [Data: Reports (615, 941)].
  • Personal Growth: Designing life, intentional living [Data: Reports (374, 1124)].

12. Ethics and Philosophy

  • Ethical Design: AI, agile methodologies [Data: Reports (1489)].
  • Philosophical Themes: Existentialism, ethics, freedom, gender roles [Data: Reports (1494)].

13. Political and Social Discourse

  • Political Rhetoric: Analysis of Donald Trump, alt-right ideology [Data: Reports (29)].
  • Social Change: Capitalism dynamics, political reform [Data: Reports (907, 1048)].

14. Arts and Culture

  • Contemporary Art: Role of museums, cultural frameworks [Data: Reports (290, 833)].
  • Music and Media: Music streaming services, audio technology [Data: Reports (1254, 1242)].

15. Space Exploration

  • Mars Exploration: NASA missions, human colonization [Data: Reports (1210)].
  • Diversity in Space Workforce: NASA’s efforts, commercial collaborations [Data: Reports (1208)].

This taxonomy provides a structured overview of the diverse topics covered in the interviews, facilitating a better understanding of the content and its implications.

Some of these are quite good, but others miss the mark. For example, I don’t understand why it’d suggest top level categories for COVID-19 or Space Exploration. Sure, these topics came up in some conversations, but these subjects aren’t central to the show.

I was also surprised with the entry about Political Rhetoric. I don’t recall discussing Donald Trump or alt-right ideology in any of my shows. When I asked the LLM about which episode focused on these topics, it admitted that there wasn’t any. But it also said this:

Guest and Speaker

The guest or speaker who discussed these topics in detail is danah. She presented a talk that included insights from an anthropologist who studied Trump voters and their Bible study practices. Additionally, she addressed the issues surrounding teenagers being drawn into alt-right rhetoric and the challenges in critical thinking education [Data: Entities (39, 41); Relationships (1954, 1955)].

Conclusion

While a specific podcast episode focusing solely on Donald Trump and alt-right ideology is not identified, the “Trump Voters and Rhetoric Analysis” community provides a thorough examination of these subjects. The insights from danah’s talk and the anthropologist’s research offer valuable perspectives on the intersection of political rhetoric, religious beliefs, and ideological influences on youth.

This is a hallucination. In episode 86, Veronica Erb discussed a study by danah boyd that delved into these subjects; I assume this is what the LLM is referencing. But I haven’t interviewed boyd, so referring to them as a “guest” is flat out wrong.

I suspect this might be an issue with the model I used to build the knowledge graph. I didn’t go with the more powerful GPT-4o, opting instead for GPT-4o Mini. The reason is cost: I initially used GPT-4o and blew through $70 or so in a few minutes. This technique requires a lot more work from the LLM than direct queries. I felt it in my wallet, but also assume the system is using a lot of energy.

So the system might work better using a more powerful model. For real (client) work, I’d likely use GPT-4o for building the knowledge graph.

Final Thoughts

Which is to say, I’m excited about using graph RAG but being a bit more mindful about where and how to apply it. It’s clearly more resource-intensive than other techniques, but allows me to do things I can’t easily do otherwise.

For information architecture, an obvious use case is producing a first draft for an organization scheme. Graph RAG allows me to take in all of the system’s content to look for patterns. The technique is much more effective than my previous experiment in this direction.

The most obvious limitation here is that the organization scheme would be based on existing content. After all, the knowledge graph would be built from what’s already there; the system wouldn’t know anything about concepts that aren’t already in the content repository.

But even here, the technique might help. Among my experiments, I asked the LLM to suggest future interviews about subjects that might be related to the things we discuss in the show. This is what it proposed:

Suggested Interview Ideas

1. Impact of Digital Tools on Reading Habits

An interview with experts on the impact of digital tools on reading habits and information retention could provide valuable insights. Platforms like Readwise, Pocket, and Goodreads have transformed how people engage with and retain information. This interview would complement existing discussions on the Reading Management Community [Data: Reports (1142)].

2. Role of Technology in Urban Planning and Accessibility

Interviewing experts on the role of technology in enhancing urban planning and accessibility, particularly for older adults and individuals with disabilities, would be highly beneficial. This would add depth to discussions on technology’s impact on urban spaces and inclusive practices [Data: Reports (973)].

3. Long-Term Effects of COVID-19

A discussion with a healthcare professional about the long-term effects of COVID-19, focusing on Long COVID and its implications for public health, would provide a comprehensive view of the pandemic’s lasting impact. This would enrich the existing coverage of COVID-19’s impact on citizens and public health measures [Data: Reports (1505)].

4. Sustainable Design Practices

Interviewing a sustainability expert to discuss the role of design in promoting sustainable practices would be a valuable addition. This would complement the existing focus on design and strategy in the Foodicons Project [Data: Reports (521)].

5. AI in Productivity Tools

An interview with a key figure from the AI-driven productivity community, such as a developer from Notion AI, could delve into the integration of AI in productivity tools and its impact on user experience. This would complement existing interviews on productivity methodologies and digital tools [Data: Reports (820)].

These interviews would provide a broader understanding of how technology, health, sustainability, and AI intersect with various aspects of daily life and professional practices, enriching the existing content with new perspectives and insights.

Pretty good! Again, I don’t know why the LLM is overindexing on COVID, but the other topics are worth considering. The first and last ones are particularly interesting; note the LLM even suggested companies I might reach out to. With this step, the LLM starts venturing into ‘collaborator’ territory.

One takeaway from this experiment: while generative AI isn’t ready to replace information architects, it’s a fabulous tool to augment our work. AIs won’t replace IAs any time soon, but they enable us to work faster, more effectively, and at greater scale. With graph RAG, we can now work on the big picture — at the level of forests, not just with individual trees.