Executive Summary

This report documents the progress of DeepCredit, our custom AI system for generating comprehensive credit rating reports. The experiment compares our improved DeepCredit v0.2 against OpenAI’s Deep Research, with both systems tasked with producing credit rating reports for Trafigura Group Pte. Ltd.

Results show that DeepCredit v0.2 achieved significant improvement over v0.1, narrowing the performance gap with OpenAI’s system. While Deep Research still maintains an advantage, DeepCredit v0.2 demonstrated marked progress in research depth, analytical quality, and rating justification. Key findings include:

• DeepCredit v0.2 earned an overall score of ~87/100 compared to Deep Research’s ~93/100
• The rating gap between systems narrowed, with DeepCredit v0.2 assigning an A (Stable) rating versus Deep Research’s BBB+ (Stable)
• Notable improvements in information quality, report structure, and rating justification

The system’s enhanced architecture, featuring improved knowledge gap identification and prioritization, better tool selection, and comprehensive logging, contributed to these gains. This experiment confirms that our agentic approach shows promise but suggests a need to reimagine the system architecture to further close the performance gap.

Background

In our initial experiment, DeepCredit v0.1 substantially underperformed OpenAI’s Deep Research, scoring approximately 70-75/100 compared to Deep Research’s 93/100. The first version employed a multi-agent architecture based on our “Agentic Deep Research” framework with specialized agents, but had limited capabilities restricted to web scraping with no PDF parsing or advanced computational functions.

This shortfall prompted a targeted redesign of DeepCredit to enhance its research capabilities, knowledge management, and decision-making processes. The goal was to narrow the performance gap with OpenAI’s system while gaining deeper insights into which architectural components most significantly impact performance in credit analysis tasks.

System Architecture Enhancements

DeepCredit v0.2 features significant architectural improvements across several dimensions, maintaining the multi-agent framework while enhancing each component’s capabilities:

1. Enhanced Knowledge Gap Agent

The Knowledge Gap Agent represents the system’s ability to identify what information is missing and prioritize research efforts accordingly. Key improvements include:

• Improved gap identification and prioritization: More sophisticated algorithms for identifying critical information gaps in financial data, corporate governance, market positioning, and risk factors
• Gap history tracking: Implementation of persistent memory to track previously attempted research directions, avoiding repetitive or unproductive paths
• Research completeness evaluation: Development of a weighted scoring system that quantifies the completeness of information across key credit rating dimensions
• Outstanding gap identification: Enhanced capability to flag areas where information remains insufficient despite research attempts

2. Upgraded Tool Selector Agent

The Tool Selector Agent determines which research tools to employ based on identified knowledge gaps. Enhancements focused on:

• Integration with Knowledge Gap Agent: Tighter coordination between gap identification and tool selection through shared context and evaluation criteria
• Improved tool selection logic: More nuanced decision-making about which research approach would be most effective for specific types of information gaps
• Previous attempt awareness: Capacity to consider the success or failure of previous tool selections when making new decisions
• Research strategy optimization: Better matching of research tools to information needs based on the nature and priority of knowledge gaps

3. Iterative Research Process Improvements

The core research workflow was redesigned to support more systematic and progressive information gathering:

• Evaluation result tracking: Implementation of a mechanism to store and utilize previous evaluation results when planning subsequent research
• Priority-based research allocation: Modified run method that allocates research effort proportionally to the criticality of different knowledge gaps
• Gap evaluation enhancement: Updated evaluation methods incorporating priority assessment and information quality metrics
• Agent selection refinement: More sophisticated agent selection considering both gap history and relative priorities
• Initialization improvements: Enhanced setup processes for evaluation tracking in the initialization method

4. New Logging System

A comprehensive logging infrastructure was implemented to track system behavior and performance:

• DeepCreditLogger class: Creation of a dedicated logging class with specialized functionality for tracking agentic research
• Multi-level logging: Implementation of differentiated log levels (HIGH_LEVEL, PROGRESS, DETAIL, ERROR, WARNING) for varied granularity
• Standardized naming convention: Establishment of consistent log file naming: YYYYMMDD_HHMMSS_MODE_COMPANYNAME.log
• Comprehensive event tracking: Detailed logging of API calls, knowledge gaps, tool selections, research results, decision processes, and performance metrics

5. Project Structure Improvements

The system’s overall architecture was enhanced for better organization and maintainability:

• Dedicated logs directory: Addition of structured storage for log files
• Updated initialization: Modified init.py to expose the new DeepCreditLogger class
• Separation of concerns: Maintenance of clear boundaries between the main entry point and utility functions

Experimental Results

The comparison between DeepCredit v0.2 and OpenAI’s Deep Research was conducted using the same evaluation criteria as our previous experiment, with both systems producing credit rating reports for Trafigura Group Pte. Ltd.

Rating Outcomes

A notable difference emerged in the final ratings:

• DeepCredit v0.2: A (Stable)
• OpenAI Deep Research: BBB+ (Stable)

This one-notch differential highlights how the systems interpreted similar information differently, with DeepCredit v0.2 taking a somewhat more optimistic view of Trafigura’s credit profile.

Performance Scores

An independent evaluation assessed both reports across three dimensions:

1. Quality and Detail of Information

• DeepCredit v0.2: 85/100
• OpenAI Deep Research: 95/100

DeepCredit v0.2 showed substantial improvement in information gathering compared to v0.1 (which scored 25/35 in the previous evaluation framework). The system effectively captured key financial metrics, discussed Trafigura’s business lines, and referenced liquidity positions. However, it still lacked some of the granularity found in the Deep Research report, particularly regarding specific risk incidents and detailed debt structure breakdowns.

2. Overall Quality of the Report

• DeepCredit v0.2: 90/100
• OpenAI Deep Research: 90/100

In this category, DeepCredit v0.2 achieved parity with Deep Research, presenting a well-structured report with clear section organization. The report featured a professional format with headings for methodology, business profile, industry factors, peer comparisons, and rating sensitivities. This represents significant progress from v0.1, which produced more of a skeletal outline than a comprehensive analysis.

3. Justification of Final Rating

• DeepCredit v0.2: 85/100
• OpenAI Deep Research: 95/100

While DeepCredit v0.2 provided clearer rating justification than v0.1, explaining the A rating based on strong liquidity, large credit facilities, diversification, and renewable energy initiatives, it still didn’t match the depth of Deep Research’s justification. The Deep Research report offered more detailed explanations of how specific financial metrics and operational risks translated into the BBB+ rating.

Overall Scores

• DeepCredit v0.2: ~87/100
• OpenAI Deep Research: ~93/100

DeepCredit v0.2 narrowed the performance gap significantly, improving from approximately 70-75/100 to ~87/100, while Deep Research maintained its high performance at ~93/100.

Key Differences in Output Quality

Strengths of DeepCredit v0.2

1. Improved Structure: The report featured well-organized sections following a conventional rating agency format, enhancing readability and professional appearance.
2. Balance of Information: The system effectively balanced quantitative data with qualitative analysis, providing context for financial metrics and industry trends.
3. Forward-Looking Analysis: DeepCredit v0.2 incorporated more forward-looking elements, including Trafigura’s strategic initiatives in renewables and green hydrogen.
4. ESG Integration: The report placed stronger emphasis on environmental sustainability factors, potentially influencing its more optimistic rating outcome.
5. Peer Comparison: Enhanced competitive analysis positioned Trafigura relative to peers like Glencore, Vitol, and Mercuria.

Remaining Gaps

1. Detail Depth: Deep Research still provided more granular financial metrics, risk incident details, and specific numeric trends.
2. Risk Assessment: Deep Research’s more comprehensive evaluation of fraud incidents, governance lapses, and event risks likely contributed to its more conservative rating.
3. Historical Context: The OpenAI system incorporated more detailed historical references and year-by-year numeric changes.
4. Rating Framework: Deep Research more explicitly linked its analysis to established rating agency methodologies and explained the rationale for not assigning an A rating.
5. External References: Deep Research included more citations to third-party sources and rating agency perspectives.

Analysis of Architectural Impact

The improvements in DeepCredit v0.2’s performance can be directly linked to specific architectural enhancements:

1. Enhanced Knowledge Gap Agent: The improved gap identification and prioritization resulted in more comprehensive coverage of essential credit rating dimensions. The system still missed some specific details but showed much better awareness of what information was needed.
2. Upgraded Tool Selector Agent: Better tool selection led to more efficient research, allowing the system to gather relevant information about Trafigura’s business operations, financial performance, and industry positioning.
3. Iterative Research Process: The refined research workflow enabled progressive information gathering, evidenced by the report’s balanced coverage across multiple dimensions of creditworthiness.
4. Logging System: While not directly affecting output quality, the comprehensive logging facilitated better understanding of the system’s decision-making processes and identified areas for further improvement.

These enhancements collectively enabled DeepCredit v0.2 to produce a more thorough, balanced, and professional credit rating report. However, the remaining performance gap suggests that certain architectural limitations still constrain the system’s capabilities.

Implications and Future Directions

Current Limitations

Despite significant improvements, DeepCredit v0.2 faces several limitations:

1. Architectural Complexity: The multi-agent system has become increasingly complex, making it difficult to isolate which components contribute most to performance gains.
2. Component Interdependence: The tight coupling between agents creates dependencies that can propagate errors or inefficiencies throughout the system.
3. Optimization Challenges: The current architecture makes it difficult to optimize individual components without affecting others.
4. Evaluation Granularity: The current evaluation framework provides useful high-level feedback but lacks the granularity needed to guide specific architectural improvements.

Planned Approach

To address these limitations and further advance DeepCredit’s capabilities, we plan to:

1. Architectural Simplification: Strip the system back to its essential components to establish a cleaner baseline.
2. Robust Evaluation Framework: Develop more granular, component-specific evaluation metrics that can isolate the impact of individual architectural elements.
3. Incremental Reconstruction: Rebuild the system one component at a time, carefully measuring the performance impact of each addition.
4. Systematic Configuration Testing: Test multiple configurations of each component to develop deeper insights into optimal settings.
5. Agentic Workflow Analysis: Focus on understanding which aspects of the agentic workflow contribute most significantly to performance and why.

This approach will enable more targeted improvements and help establish a clearer understanding of the architectural principles that drive performance in AI-powered credit analysis.

Conclusion

DeepCredit v0.2 represents a significant advancement in our AI-powered credit analysis capabilities. The narrowing performance gap with OpenAI’s Deep Research demonstrates the potential of our agentic approach while highlighting areas for continued improvement.

The experiment confirms several key insights:

1. Multi-agent architectures can effectively approach complex analytical tasks like credit rating, but require careful design and coordination.
2. Systematic knowledge gap identification and prioritization significantly impact research quality and comprehensiveness.
3. Different architectural approaches can lead to divergent analytical conclusions, as evidenced by the one-notch rating difference.

While DeepCredit v0.2 shows promise, its increasing complexity suggests that continued incremental improvements may yield diminishing returns. The planned redesign, focusing on systematic evaluation and component-by-component optimization, offers a more promising path toward closing the remaining performance gap with state-of-the-art systems.

By developing deeper insights into which aspects of the agentic workflow contribute most to performance, we aim to establish a more efficient and effective architecture for AI-powered credit analysis that could potentially exceed the capabilities of current leading systems.