MLflow 集成

MLflow 概览

MLflow 是一个开源平台，用于帮助机器学习从业者和团队应对机器学习流程中的复杂性。

它提供追踪功能，通过捕获应用服务执行的详细信息来增强生成式 AI 应用中的 LLM 可观测性。追踪提供了一种记录请求每个中间步骤的输入、输出和元数据的方法，让你能够轻松定位 bug 和意外行为的来源。

MLflow crewAI 追踪使用概览

功能

追踪仪表盘：通过包含 span 的输入、输出和元数据的详细仪表盘监控 crewAI 代理的活动。
自动化追踪：与 crewAI 的完全自动化集成，只需运行 mlflow.crewai.autolog() 即可启用。
仅需少量工作量的手动追踪埋点：通过 MLflow 的高层 fluent API（如装饰器、函数包装器和上下文管理器）自定义追踪埋点。
OpenTelemetry 兼容性：MLflow Tracing 支持将追踪导出到 OpenTelemetry Collector，随后可导出到 Jaeger、Zipkin 和 AWS X-Ray 等后端。
打包并部署代理：将你的 crewAI 代理打包并部署到推理服务器，并支持多种部署目标。
安全托管 LLM：通过 MFflow gateway 在一个统一端点中托管来自多个提供商的多个 LLM。
评估：使用便捷的 mlflow.evaluate() API 和广泛的指标集评估你的 crewAI 代理。

设置说明

安装 MLflow 包

# crewAI 集成在 mlflow>=2.19.0 中可用
pip install mlflow

启动 MFflow 跟踪服务器

# 该过程是可选的，但建议使用 MLflow tracking server 以获得更好的可视化和更丰富的功能。
mlflow server

在应用中初始化 MLflow

将下面两行添加到你的应用代码中：

import mlflow

mlflow.crewai.autolog()

# 可选：如果你有 tracking server，可以设置 tracking URI 和 experiment 名称
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("CrewAI")

用于追踪 CrewAI 代理的示例用法：

from crewai import Agent, Crew, Task
from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource
from crewai_tools import SerperDevTool, WebsiteSearchTool

from textwrap import dedent

content = "Users name is John. He is 30 years old and lives in San Francisco."
string_source = StringKnowledgeSource(
    content=content, metadata={"preference": "personal"}
)

search_tool = WebsiteSearchTool()


class TripAgents:
    def city_selection_agent(self):
        return Agent(
            role="City Selection Expert",
            goal="Select the best city based on weather, season, and prices",
            backstory="An expert in analyzing travel data to pick ideal destinations",
            tools=[
                search_tool,
            ],
            verbose=True,
        )

    def local_expert(self):
        return Agent(
            role="Local Expert at this city",
            goal="Provide the BEST insights about the selected city",
            backstory="""A knowledgeable local guide with extensive information
        about the city, it's attractions and customs""",
            tools=[search_tool],
            verbose=True,
        )


class TripTasks:
    def identify_task(self, agent, origin, cities, interests, range):
        return Task(
            description=dedent(
                f"""
                Analyze and select the best city for the trip based
                on specific criteria such as weather patterns, seasonal
                events, and travel costs. This task involves comparing
                multiple cities, considering factors like current weather
                conditions, upcoming cultural or seasonal events, and
                overall travel expenses.
                Your final answer must be a detailed
                report on the chosen city, and everything you found out
                about it, including the actual flight costs, weather
                forecast and attractions.

                Traveling from: {origin}
                City Options: {cities}
                Trip Date: {range}
                Traveler Interests: {interests}
            """
            ),
            agent=agent,
            expected_output="Detailed report on the chosen city including flight costs, weather forecast, and attractions",
        )

    def gather_task(self, agent, origin, interests, range):
        return Task(
            description=dedent(
                f"""
                As a local expert on this city you must compile an
                in-depth guide for someone traveling there and wanting
                to have THE BEST trip ever!
                Gather information about key attractions, local customs,
                special events, and daily activity recommendations.
                Find the best spots to go to, the kind of place only a
                local would know.
                This guide should provide a thorough overview of what
                the city has to offer, including hidden gems, cultural
                hotspots, must-visit landmarks, weather forecasts, and
                high level costs.
                The final answer must be a comprehensive city guide,
                rich in cultural insights and practical tips,
                tailored to enhance the travel experience.

                Trip Date: {range}
                Traveling from: {origin}
                Traveler Interests: {interests}
            """
            ),
            agent=agent,
            expected_output="Comprehensive city guide including hidden gems, cultural hotspots, and practical travel tips",
        )


class TripCrew:
    def __init__(self, origin, cities, date_range, interests):
        self.cities = cities
        self.origin = origin
        self.interests = interests
        self.date_range = date_range

    def run(self):
        agents = TripAgents()
        tasks = TripTasks()

        city_selector_agent = agents.city_selection_agent()
        local_expert_agent = agents.local_expert()

        identify_task = tasks.identify_task(
            city_selector_agent,
            self.origin,
            self.cities,
            self.interests,
            self.date_range,
        )
        gather_task = tasks.gather_task(
            local_expert_agent, self.origin, self.interests, self.date_range
        )

        crew = Crew(
            agents=[city_selector_agent, local_expert_agent],
            tasks=[identify_task, gather_task],
            verbose=True,
            memory=True,
            knowledge={
                "sources": [string_source],
                "metadata": {"preference": "personal"},
            },
        )

        result = crew.kickoff()
        return result


trip_crew = TripCrew("California", "Tokyo", "Dec 12 - Dec 20", "sports")
result = trip_crew.run()

print(result)

参考 MLflow Tracing 文档了解更多配置和使用场景。

可视化代理活动

现在，你的 crewAI 代理追踪已被 MLflow 捕获。接下来访问 MLflow tracking server 来查看追踪并获取对代理的洞察。
在浏览器中打开 127.0.0.1:5000 访问 MLflow tracking server。

MLflow 追踪仪表盘